Natural language processing to merge related alert messages for accessibility

ABSTRACT

A method for merging incoming alerts for accessibility is described. A first input alert and a second input alert intended for presentation by a screen reader are received. If the first input alert and the second input alert have arrived with a specified time interval, the first input alert and the second input alert are combined into an output alert. The output alert is sent to a screen reader for presentation.

BACKGROUND OF THE INVENTION

This disclosure relates generally to natural language processing. Moreparticularly, it relates to using natural language processing to mergerelated alert messages for accessibility.

It is known for a computer to provide certain hardware and softwaretechnologies for “accessibility” that help visually or physicallyimpaired people to use a computer. For example, the AccessibilityOptions control panel in the Windows™ operating system provideskeyboard, mouse and screen options for people who have difficulty typingor seeing the screen. These technologies are also called assistivetechnologies.

One such program is a screen reader for the visually impaired. Screenreaders are software programs that will provide synthesized speech toenable visually impaired users to use a computer operation system orcomputer applications. Typically, a screen reader will read textpresented in one or more displays areas on the computer screen. A screenreader will also provide an alert notification when a user has taken anaction. For example, if a user is attempting to purchase items from anonline store, and pushes a button to add an item to a virtual cart, thescreen reader will alert the user via synthesized speech that XYZ itemwas added to the cart.

One problem encountered with screen readers is that if the user takesactions too quickly, the screen reader does not provide the informationaccurately. For example, if a user wanted one XYZ item and one ABC item,but accidentally pressed the XYZ button twice, then the screen readermay either interrupt the first notification with a second notificationor play two identical notifications in quick succession. The result isthat the user is often unsure of what occurred and how to proceed.

This present invention provides a solution to this problem.

BRIEF SUMMARY

According to this disclosure, a method for merging incoming alerts foraccessibility is described. A first input alert and the second inputalert intended for presentation by a screen reader are received. If thefirst input alert and the second input alert have arrived with aspecified time interval, the first input alert and the second inputalert are combined into an output alert. The output alert is sent to ascreen reader for presentation.

The foregoing has outlined some of the more pertinent features of thedisclosed subject matter. These features should be construed to bemerely illustrative. Many other beneficial results can be attained byapplying the disclosed subject matter in a different manner or bymodifying the invention as will be described.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 depicts an exemplary block diagram of a distributed dataprocessing environment in which exemplary aspects of the illustrativeembodiments may be implemented;

FIG. 2 is an exemplary block diagram of a data processing system inwhich exemplary aspects of the illustrative embodiments may beimplemented;

FIG. 3 illustrates a set of computer applications resident in memory ofa computer system which can implement embodiments of the invention;

FIG. 4 illustrates a user interface of a web application which isenhanced with assistive technology;

FIG. 5 illustrates a set of flows between elements in one embodiment ofthe invention;

FIG. 6 is a flow diagram of creating an output alert according to anembodiment of the invention; and

FIG. 7 is a flow diagram of examining the dependency trees of two inputalerts to create an output alert according to an embodiment of theinvention.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

At a high level, preferred embodiments of the invention provide animprovement to screen reader technology that incorporates naturallanguage processing (NLP) to merge input alerts into an output alertwhen appropriate. Similar elements of similar input alerts are mergedinto a single output alert, while dissimilar input alerts areconcatenated into a single output alert. For example, if the user clicksto add one XYZ item and then one ABC item to the cart, instead ofplaying two messages by the screen reader, “You have added XYZ to yourcart” and “You have added ABC to your cart”, the NLP processing willidentify these as the same type of alert and merge them. For example, amerged alert in embodiments of the invention will play, “You have addedan XYZ and an ABC to your cart.” By merging alerts, there is a reducedchance that one alert will play over another. In addition, the mergedalert will often summarize the present state of the application moreeffectively for a listener than a series of alerts as the listener willnot need to mentally keep track of a long list of actions.

With reference now to the drawings and in particular with reference toFIGS. 1-2, exemplary diagrams of data processing environments areprovided in which illustrative embodiments of the disclosure may beimplemented. It should be appreciated that FIGS. 1-2 are only exemplaryand are not intended to assert or imply any limitation with regard tothe environments in which aspects or embodiments of the disclosedsubject matter may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

With reference now to the drawings, FIG. 1 depicts a pictorialrepresentation of an exemplary distributed data processing system inwhich aspects of the illustrative embodiments may be implemented.Distributed data processing system 100 may include a network ofcomputers in which aspects of the illustrative embodiments may beimplemented. The distributed data processing system 100 contains atleast one network 102, which is the medium used to provide communicationlinks between various devices and computers connected together withindistributed data processing system 100. The network 102 may includeconnections, such as wire, wireless communication links, or fiber opticcables.

In the depicted example, server 104 and server 106 are connected tonetwork 102 along with storage unit 108. In addition, clients 110, 112,and 114 are also connected to network 102. These clients 110, 112, and114 may be, for example, personal computers, network computers, or thelike. In the depicted example, server 104 provides data, such as bootfiles, operating system images, and applications to the clients 110,112, and 114. Clients 110, 112, and 114 are clients to server 104 in thedepicted example. Distributed data processing system 100 may includeadditional servers, clients, and other devices not shown.

In the depicted example, distributed data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 1 is intended as anexample, not as an architectural limitation for different embodiments ofthe disclosed subject matter, and therefore, the particular elementsshown in FIG. 1 should not be considered limiting with regard to theenvironments in which the illustrative embodiments of the presentinvention may be implemented.

With reference now to FIG. 2, a block diagram of an exemplary dataprocessing system is shown in which aspects of the illustrativeembodiments may be implemented. Data processing system 200 is an exampleof a computer, such as client 110 in FIG. 1, in which computer usablecode or instructions implementing the processes for illustrativeembodiments of the disclosure may be located.

With reference now to FIG. 2, a block diagram of a data processingsystem is shown in which illustrative embodiments may be implemented.Data processing system 200 is an example of a computer, such as server104 or client 110 in FIG. 1, in which computer-usable program code orinstructions implementing the processes may be located for theillustrative embodiments. In this illustrative example, data processingsystem 200 includes communications fabric 202, which providescommunications between processor unit 204, memory 206, persistentstorage 208, communications unit 210, input/output (I/O) unit 212, anddisplay 214.

Processor unit 204 serves to execute instructions for software that maybe loaded into memory 206. Processor unit 204 may be a set of one ormore processors or may be a multi-processor core, depending on theparticular implementation. Further, processor unit 204 may beimplemented using one or more heterogeneous processor systems in which amain processor is present with secondary processors on a single chip. Asanother illustrative example, processor unit 204 may be a symmetricmulti-processor (SMP) system containing multiple processors of the sametype.

Memory 206 and persistent storage 208 are examples of storage devices. Astorage device is any piece of hardware that is capable of storinginformation either on a temporary basis and/or a permanent basis. Memory206, in these examples, may be, for example, a random access memory orany other suitable volatile or non-volatile storage device. Persistentstorage 208 may take various forms depending on the particularimplementation. For example, persistent storage 208 may contain one ormore components or devices. For example, persistent storage 208 may be ahard drive, a flash memory, a rewritable optical disk, a rewritablemagnetic tape, or some combination of the above. The media used bypersistent storage 208 also may be removable. For example, a removablehard drive may be used for persistent storage 208.

Communications unit 210, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 210 is a network interface card. Communications unit210 may provide communications through the use of either or bothphysical and wireless communications links.

Input/output unit 212 allows for input and output of data with otherdevices that may be connected to data processing system 200. Forexample, input/output unit 212 may provide a connection for user inputthrough a keyboard and mouse. Further, input/output unit 212 may sendoutput to a printer. Display 214 provides a mechanism to displayinformation to a user.

Instructions for the operating system and applications or programs arelocated on persistent storage 208. These instructions may be loaded intomemory 206 for execution by processor unit 204. The processes of thedifferent embodiments may be performed by processor unit 204 usingcomputer implemented instructions, which may be located in a memory,such as memory 206. These instructions are referred to as program code,computer-usable program code, or computer-readable program code that maybe read and executed by a processor in processor unit 204. The programcode in the different embodiments may be embodied on different physicalor tangible computer-readable media, such as memory 206 or persistentstorage 208.

Program code 216 is located in a functional form on computer-readablemedia 218 that is selectively removable and may be loaded onto ortransferred to data processing system 200 for execution by processorunit 204. Program code 216 and computer-readable media 218 form computerprogram product 220 in these examples. In one example, computer-readablemedia 218 may be in a tangible form, such as, for example, an optical ormagnetic disc that is inserted or placed into a drive or other devicethat is part of persistent storage 208 for transfer onto a storagedevice, such as a hard drive that is part of persistent storage 208. Ina tangible form, computer-readable media 218 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory that is connected to data processing system 200. The tangibleform of computer-readable media 218 is also referred to ascomputer-recordable storage media. In some instances,computer-recordable media 218 may not be removable.

Alternatively, program code 216 may be transferred to data processingsystem 200 from computer-readable media 218 through a communicationslink to communications unit 210 and/or through a connection toinput/output unit 212. The communications link and/or the connection maybe physical or wireless in the illustrative examples. Thecomputer-readable media also may take the form of non-tangible media,such as communications links or wireless transmissions containing theprogram code. The different components illustrated for data processingsystem 200 are not meant to provide architectural limitations to themanner in which different embodiments may be implemented. The differentillustrative embodiments may be implemented in a data processing systemincluding components in addition to or in place of those illustrated fordata processing system 200. Other components shown in FIG. 2 can bevaried from the illustrative examples shown. As one example, a storagedevice in data processing system 200 is any hardware apparatus that maystore data. Memory 206, persistent storage 208, and computer-readablemedia 218 are examples of storage devices in a tangible form.

In another example, a bus system may be used to implement communicationsfabric 202 and may be comprised of one or more buses, such as a systembus or an input/output bus. Of course, the bus system may be implementedusing any suitable type of architecture that provides for a transfer ofdata between different components or devices attached to the bus system.Additionally, a communications unit may include one or more devices usedto transmit and receive data, such as a modem or a network adapter.Further, a memory may be, for example, memory 206 or a cache such asfound in an interface and memory controller hub that may be present incommunications fabric 202.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJava™, Smalltalk, C++, C#, Objective-C, or the like, and conventionalprocedural programming languages such as Python or C. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer, or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 1-2 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 1-2. Also, theprocesses of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thedisclosed subject matter.

As will be seen, the techniques described herein may operate inconjunction within the standard client-server paradigm such asillustrated in FIG. 1 in which client machines communicate with anInternet-accessible Web-based portal executing on a set of one or moremachines. End users operate Internet-connectable devices (e.g., desktopcomputers, notebook computers, Internet-enabled mobile devices, or thelike) that are capable of accessing and interacting with the portal.Typically, each client or server machine is a data processing systemsuch as illustrated in FIG. 2 comprising hardware and software, andthese entities communicate with one another over a network, such as theInternet, an intranet, an extranet, a private network, or any othercommunications medium or link. A data processing system typicallyincludes one or more processors, an operating system, one or moreapplications, and one or more utilities.

Screen readers are applications which help make the visual content ofthe computer screen accessible to blind or visually impaired people.Screen readers work closely with the computer's operating system (OS) toprovide information about icons, menus, dialogue boxes, files andfolders. A Text-To-Speech (TTS) engine is used to translate on-screeninformation as well as user actions into speech, which can be heard,i.e. presented, through earphones or speakers. A visually impairedcomputer user typically uses a combination of screen reader commands andoperating system commands to accomplish computer tasks. One common wayfor a screen reader to receive input is through an “alert”.

FIG. 3 is an exemplary illustration of the applications which would beresident in a computer capable of implementing the invention. Thecomputer itself is similar to that described in association with FIGS.1-2. In computer memory 300, a screen reader application 301 isresident. A natural language processing module 303, e.g., a naturallanguage parser, is also resident, either as part of the screen reader301 or as a separate application. A listener module 304 interceptsalerts intended for the screen reader, and if two or more alerts arereceived in quick succession, forwards them to the NLP module 303 forhandling. A web browser 305 and a plurality of other applications 307are also resident. The user interacts with the web browser 305 andapplications 307 to perform tasks on the computer under the control ofthe operating system 309. The web browser 305 and the web pages that itinteracts with, the applications 307 and the operating system 309 areall potential sources for alerts. In preferred embodiments of theinvention, the web browser 305, at least some of the applications 307and the operating system are “accessible” or “accessibility aware”programs. In embodiments of the invention, the web pages with which theweb browser interacts are configured to send alerts to the screen readerwhen the user performs actions on an element in the web page.

Accessible web applications and web pages use alerts that screen readersuse to alert the user to actions which the application and user havetaken. For example, referring to the highly simplified illustration inFIG. 4 of a typical e-commerce application 401, an alert might begenerated with the text “You have added one Lion King DVD to your cart”if the user clicks a mouse button while mouse cursor 403 is over icon405. In a discovery UI application, an alert might be generated with thetext “You have selected ‘Wikipedia’ as a source filter.”

Screen readers that naively read these messages in order or thatinterrupt currently playing messages risk losing or duplicatinginformation. For example, if a user accidentally clicks on a “buy”button 405 twice in an e-commerce application 401, the screen readermight say “You have added one Lion King DVD to your cart . . . you haveadded one Lion King DVD to your cart”, leaving the user confused aboutif they added two DVDs, or whether the application just sent the alerttwice. Even worse, the screen reader might interrupt and truncate thefirst message, saying “You . . . You have added 1 Lion King DVD to yourcart” which does not accurately describe what has happened.

Embodiments of the invention use natural language processing (NLP) andalert generation techniques to merge related accessibility alerts.Continuing the e-commerce example, suppose the user buys two items inquick succession, for example, by mouse clicking over icon 403 and thenicon 409. In this case, the e-commerce web application would generatetwo alerts: “You have added one Lion King DVD to your cart” and “Youhave added one Pocahontas DVD to your cart”. In this example,embodiments of the invention would recognize the similarity between thetwo alerts and merge them into a single alert: “You have added one LionKing DVD and one Pocahontas DVD to your cart”. This eliminates thepossibility of cutting a first alert off to start a second one.Alternately, if a user accidentally clicks the “buy” button 403 twice,the prior art e-commerce web application would generate two alerts: “Youhave added one Lion King DVD to your cart” and “You have added one LionKing DVD to your cart”. In this example, the inventive system wouldmerge these alerts into a single alert saying, “You have added two LionKing DVDs to your cart.”

In preferred embodiments of the invention, a listener module interceptsalerts intended for the screen reader. In this illustrative example, andreferring to FIG. 5, there are two Alerts, represented as a list ofstrings, A and B. These alerts are intercepted by the listener moduleand represented by flows 501 and 502. After determining that the stringshave arrived within a predetermined time interval, the listener feedsthese strings to a natural language parser, flow 503. Embodiments of theinvention use a dependency-style slot grammar parser, like the EnglishSlot Grammar (ESG) parser used in IBM Watson. The dependency-styleparsers are advantageous in that they are capable of determiningrelationships between parts of the alert, such as agent, direct objectand indirect object relationships. Constituent-style parsers can be usedin alternative embodiments of the invention, however, another modulewill need to be added to figure out the relationships between parts ofthe alert.

The parser will start at the root node (probably the matrix verb of asimple declarative sentence), to evaluate if the root node in alert A isequivalent to the root node in alert B. In one embodiment, equivalenceor sufficient similarity between alerts is defined as having anequivalent headword (the same word or a synonym) and having the same setof outgoing link labels. “Equivalence”, “similarity” and “sufficientsimilarity” are used interchangeably in the specification to mean “closeenough” according to a similarity rule. In one embodiment, equivalenceor sufficient similarity of the dependency as a whole may depend on thepart of speech of the headword, or root node, enabling merged alertslike “You have added one copy of ‘The Lion King’ and one copy of‘Pocahontas’ to your cart”. Having root or child nodes that areidentical is not required for node equivalence. Similarity rules areused in embodiments of the invention to determine whether the dependencytrees are similar as well as determining whether corresponding nodes ofthe respective trees are similar. If the root nodes in the two alertsare equivalent, then the parser recursively steps down the dependencytree for each alert. In embodiments of the invention, equivalence orsubstantial similarity is established by comparing the dependency treesof the two alerts. If the two dependency trees are equivalent orsubstantially similar according to a rule then the alerts will bemerged. If not, the strings in the non-equivalent alerts are joinedusing a coordination operation. One possible definition of acoordination operation is defined as concatenation of the two alerts,e.g., joined by “and” or another conjunction.

The parser returns the merged or concatenated alert back to the listenerin flow 504. In flow 505, the listener module forwards the merged orconcatenated alert to the screen reader which will present it to theuser.

A more detailed example of the process in one embodiment is illustratedin FIG. 6 and described below. First, the listener module receivesalerts from a web application, step 601. In this example, there are twoalerts:

Alert A: “You have added a Lion King DVD to your cart.”Alert B: “You have added a Pocahontas DVD to your cart.”

In step 603, the listener module determines whether the two alerts havecome in a predetermined time interval. The predetermined time intervalis preferably set so that alerts will not “collide”, i.e. have one alertinterrupt the other. In embodiments of the invention, the predeterminedtime interval is the range of 2-15 seconds. In an alternativeembodiment, the predetermined time interval, or collision interval isdynamic and depends on the length of the first incoming alert. If thefirst alert is longer, i.e. taking longer to present, the collisioninterval will also be longer. If the two alerts have come in within thepredetermined time interval, they are passed to the parser for analysisin step 605. If not, they are passed to the screen reader forpresentation in step 613.

In step 605, the parser begins parsing the alerts. It starts buildingdependency trees for each alert (step 607) with the root node of eachalert and determines the relationships between nodes. In preferredembodiments of the invention, the dependency tree is used to determineequivalence between alerts as well as providing information for a mergeoperation for equivalent alerts. In this example, both input alerts havea root node of “add” and have respective agent (subj), direct object(do) and indirect object (io) relationships between respective nodes ofthe alerts. The dependency trees are determined to be equivalent, step609, so the parser begins the process of merging the alerts in step 611.At a high level, the merge step determines which parts of the inputalerts should be part of the output merged alert. One preferred methodfor merging the alerts is described with reference to FIG. 7. The mergedalert is sent to the screen reader for presentation to the user, step613.

If the dependency trees were not judged equivalent or sufficientlysimilar, in embodiments of the invention, the process proceeds to step615. Instead of determining which parts of the input alerts should bepart of the merged alert, in this case, the two input alerts areconcatenated together, using a conjunction like “and” to join the twoalerts into a single output alert. Assume that instead of Alert B, anAlert C was intercepted “You are now in checkout.” Also assume that theparser determined that Alert A and Alert C were not sufficiently similarto merge them. In this case, the parser would proceed to step 615 andcreate the concatenated alert ““You have added a Lion King DVD to yourcart, and you are now in checkout.”

Next, the concatenated alert would be sent to the screen reader forpresentation to the user, step 613.

A more detailed flow diagram of some of the steps from FIG. 6 isillustrated in FIG. 7. In step 701, the parser begins parsing the alertsto build the first dependency tree for each alert. Keeping with theexample above, with the root node of each alert, the parser determinesthe relationships between nodes. In this example, both alerts have aroot node of “add” and have respective agent (subj), direct object (do)and indirect object (io) relationships between respective nodes of thealerts. To determine equivalency, in some embodiments of the invention,the full dependency tree is not needed, only enough of the tree todetermine whether the equivalency rule is satisfied.

Exemplary equivalence rules in determining node equivalence used inembodiments of the invention include:

A. An exact string match of the wordB. An exact string match of the word and an exact match of thepart-of-speechC. A match of two words that are synonyms, e.g., “You have added a LionKing Videotape” and “You have added a Pocahontas VHS”D. Near string matches, to account for typos

Another type of equivalency rule is for determining whether the treesthemselves are equivalent, e.g., the root node is the same or equivalentand a specified number or percentage of the sub-nodes are the same orequivalent. As another example, an equivalency rule for determiningtrees are equivalent in embodiments of the invention would be the rootnode is the same or equivalent and a second type of node is the same orequivalent.

Next, according to the equivalency rule, the two alerts are determinedto be equivalent in step 703.

To determine which of the elements from the two input alerts should beselected for the output merged alert, in some embodiments, a fulldependency tree is built, and the elements and their position in thedependency tree are compared for equivalence. In the example, the parserrecursively steps down to each of the nodes attached to the outgoinglinks. Object preposition (objprep) relationships are also determined insome embodiments of the invention. Outgoing links are part of thesyntactic model used in preferred embodiments of the invention. Alllinks between words are directed. For example, there is a link goingfrom the word “have” to “you”. The link is directed from “have” to“you”. So “have” has an outgoing link called “subj” going to “you”. Inthe same way, “you” has an incoming link called “subj” coming from“have”. This recursion provides information needed in the mergeoperation, step 715.

Next, in step 705, the parser follows the agent or “subj” relationshipin both alerts. In both alerts, the “you” nodes are equivalent and aredetermined to be equivalent by the parser. The “you” nodes in each alerthas no outgoing links so the recursion stops here in this branch of thedependency tree.

Next, in step 707, the parser follows the direct object or “do”relationship in both alerts. In this example, “Lion King DVD” is notequivalent to “Pocahontas DVD”, so these nodes are stored to be combinedvia a coordination operation. The coordination operation means tocombine the two phrases with the word “and”. For example, “You haveadded an apple to your cart”+“You have added a banana to your cart”=“Youhave added an apple and a banana to your cart”.

Then, in step 709, the parser follows the indirect object or “io”relationship in both alerts. The node is “to” in both cases, and eachnode has an “objprep” relationship. These nodes are equivalent and theparser continues to the next level of the dependency tree. The parserfollows the “objprep” relation for both alerts, step 711. In both cases,the node is “cart” and has a “det” relationship. The set (or“determiner”) relationship is the name of the connection between a nounand the determiner (also called “article”) that comes before it. Forexample, the relationship between “an” and “apple” in “You have added anapple to your cart”. These nodes are equivalent and the parser continuesto the next level.

The parser follows the “det” relationship for both alerts in step 713.The “your” nodes are deemed equivalent and are combined. Next, in step715, the merged alert is constructed. The elements or nodes which weredeemed equivalent are placed in the merged alert in the order in whichthey occurred in the input alerts. The nodes which were deemed notequivalent (or similar) are not merged, but combined as they exist inthe input alerts in the merged alert. In preferred embodiments, aconjunction such as “and” is added between the two dissimilar elements.The merged alert of this example is “You have added one Lion King DVDand one Pocahontas DVD to your cart”. Where elements meet a similarityrule, but are not identical, one of the elements is chosen for inclusionin the merged alert. In alternative embodiments, a predetermined synonymfor a set of elements found in the input alerts is used in the mergedoutput alert.

Another feature of the invention is where there is a number associatedwith similar elements in each alerts, the parser will sum the number ofelements in the output alert presented to the user.

A developer of a web page can implement a service that listens forevents which correspond to alerts. In a web page, an alert or alert roleis added to new or existing web page elements to provide alerts to theuser. Examples of alerts include: “An invalid value was entered intocredit card field” and “your session with ABC company is about toexpire”. An alert can be added statically or dynamically into the HTMLcode by the developer. Alternatively, a script library can be used. ARIAis a standard which specifies how to include accessibility content inweb pages, particularly dynamic user interface components usingprogramming languages such as Ajax, HTML and JavaScript. Web developersincreasingly use client-side scripts to provide user interface controls;ARIA describes how to add content such as an alert to make the UIcontrols accessible to a visually impaired user. The web page developercan add accessibility tags to web page content so that a screen readerwill know which portions of the web page to read.

In embodiments of the invention, when an alert event is received by thelistener, a configurable timer begins to count down. If the timerreaches 0, the text associated with the alert event is provided to theinterface, e.g., a JavaScript Document Object Model (DOM) interface,which allows the screen reader to read it as an alert. If an additionalalert event is received by the listener before the timer reaches 0, thenew alert event's text is combined with the older alert event's text asdescribed above and the timer is reset and begins to count down again.The timer is reset once the alerts are combined since if another alertcomes into the listener before the timer reaches 0, it is then combinedwith the other input alerts via the same process. Once the timer reaches0, the combined alert events' texts are provided in HTML to theinterface that allows the screen reader to read them as a single alert.In preferred embodiments of the invention, each alert event can beconfigured to be allowed or suppressed, if it is a duplicate of an alertthat is already in the listener.

When the input alerts are judged dissimilar according to the similarityrule, embodiments of the invention will concatenate rather than mergethe input alerts to form the output alert. As an example, first inputalert “you transferred $10 from your checking account to Kyle” andsecond input alert “you have selected a transfer checkbox” aredissimilar. Embodiments of the invention would create a concatenatedalert “you transferred $10 from your checking account to Kyle and youhave selected a transfer checkbox”.

As an additional example of a merged alert consider the input alert “youtransferred $500 from your spending to your checking account” and theinput alert “you transferred $750 from your savings to your checkingaccount”. This could result in the output alert of “you transferred $500from your spending to your checking account and $750 from your savingsto your checking account”. In this case, there were more dissimilarelements which needed to be included in the merged alert, but thesimilar elements “you” and “transferred” have been merged to occur asingle time.

Another example of an output merged alert receives a first input alert“you transferred 500 dollars from your savings to your checking account”and a second input alert “you transferred 750 dollars from your savingsto your checking account” within the predetermined time period. Thiscould result in the merged alert “you transferred 1250 dollars from yoursavings to your checking account”. This example shows merging quantitiesof similar elements in the input alerts to produce a single mergedelement.

An example of a syntactic/semantic parse tree (dependency tree) usingthe input alert “You have added one Pocahontas DVD to your cart” is asfollows:

------------------------------------------------------------------------.------- subj(n)  you(1)   noun pron pl def h perspron anim o------- top  have_perf(2,1,3) verb vfin vpres pl vsubj badvenadj auxv --------auxcomp(ena) add1(3,1,6,u,7) verb ven  | .--- nadj  one1(4,u)   noun numsg  ′----- obj(n)  Pocahantas DVD(6) noun propn sg glom notfnd unkph ′----- comp(p)  to2(7,3,9)  prep pprefv motionp  | .- ndet   your1(8)  det sg possdet ingdet  ′--- objprep(n) cart1(9)   noun cn sg------------------------------------------------------------------------The diagram represents a syntactic dependency tree, which is recognizedby those skilled in the art.

Here is an example of a pseudo-code implementation of a listenerfunction. In preferred embodiments of the invention, the listenerfunction would be implemented in JavaScript.

Listener:

alertCount = 0; alertList = [ ]; When(‘alert’, AlertTriggered( ));AlertTriggered = function(newAlert, allowDuplicateAlerts) {   showAlert= false;   if(alertCount == 0) {     alertText = “”;   }  if(allowDuplicateAlerts || !alertList.includes(newAlert))) {    alertCount++;     alertList.add(newAlert);     currentAlertCount =alertCount;     alertText = combineAlerts(alertText, newAlert);    Wait(function( ) {       if(currentAlertCount == alertCount) {        showAlert = true;         alertCount = 0;         alertList = [];       }     }, delayTime);   } }

In embodiments of the invention, the listener function listens foralerts and sends them to be combined by the parser if appropriate. Thelistener function can be triggered by the user. The following ispseudo-code for triggering the function.

Triggering an alert event:

TriggerEvent(‘alert’, alertText); TriggerEvent(‘alert’, alertText,true); TriggerEvent(‘alert’, alertText, false);

In alternative embodiments of the invention, rather than having aseparate listener module to intercept alerts intended for the screenreader and an NLP parser module to determine the similarity of thealerts and merge the alerts, these capabilities are built into a screenreader.

The present invention has advantages over the prior art. Rather thanallow two alerts to collide in the screen reader, the present inventionintercepts a plurality of input alerts and combines them into a singleoutput alert for presentation by the screen reader. Where the inputalerts satisfy a similarity rule, the two input alerts are merged into amerged alert. Where the input alerts do not satisfy the similarity rule,the two input alerts are concatenated into the output alert. By mergingalerts, there is a reduced chance that one alert will play over anotheralert. In addition, the merged alert will often summarize the presentstate of the application more effectively for a listener than a seriesof alerts as the listener will not need to mentally keep track of a longlist of actions.

While a preferred operating environment and use case has been described,the techniques herein may be used in any other operating environment inwhich it is desired to deploy services.

As has been described, the functionality described above may beimplemented as a standalone approach, e.g., one or more software-basedfunctions executed by one or more hardware processors, or it may beavailable as a managed service (including as a web service via aSOAP/XML or RESTful interface). The particular hardware and softwareimplementation details described herein are merely for illustrativepurposes are not meant to limit the scope of the described subjectmatter.

More generally, computing devices within the context of the disclosedsubject matter are each a data processing system comprising hardware andsoftware, and these entities communicate with one another over anetwork, such as the Internet, an intranet, an extranet, a privatenetwork, or any other communications medium or link. The applications onthe data processing system provide native support for Web and otherknown services and protocols including, without limitation, support forHTTP, FTP, SMTP, SOAP, XML, WSDL, UDDI, and WSFL, among others.Information regarding SOAP, WSDL, UDDI and WSFL is available from theWorld Wide Web Consortium (W3C), which is responsible for developing andmaintaining these standards; further information regarding HTTP, FTP,SMTP and XML is available from Internet Engineering Task Force (IETF).

In addition to the cloud-based environment, the techniques describedherein may be implemented in or in conjunction with various server-sidearchitectures including simple n-tier architectures, web portals,federated systems, and the like.

Still more generally, the subject matter described herein can take theform of an entirely hardware embodiment, an entirely software embodimentor an embodiment containing both hardware and software elements. In apreferred embodiment, the module functions are implemented in software,which includes but is not limited to firmware, resident software,microcode, and the like. Furthermore, the interfaces and functionalitycan take the form of a computer program product accessible from acomputer-usable or computer-readable medium providing program code foruse by or in connection with a computer or any instruction executionsystem. For the purposes of this description, a computer-usable orcomputer readable medium can be any apparatus that can contain or storethe program for use by or in connection with the instruction executionsystem, apparatus, or device. The medium can be an electronic, magnetic,optical, electromagnetic, infrared, or a semiconductor system (orapparatus or device). Examples of a computer-readable medium include asemiconductor or solid state memory, magnetic tape, a removable computerdiskette, a random access memory (RAM), a read-only memory (ROM), arigid magnetic disk and an optical disk. Current examples of opticaldisks include compact disk-read only memory (CD-ROM), compactdisk-read/write (CD-R/W) and DVD. The computer-readable medium is atangible, non-transitory item.

The computer program product may be a product having programinstructions (or program code) to implement one or more of the describedfunctions. Those instructions or code may be stored in a computerreadable storage medium in a data processing system after beingdownloaded over a network from a remote data processing system. Or,those instructions or code may be stored in a computer readable storagemedium in a server data processing system and adapted to be downloadedover a network to a remote data processing system for use in a computerreadable storage medium within the remote system.

In a representative embodiment, the techniques are implemented in aspecial purpose computing platform, preferably in software executed byone or more processors. The software is maintained in one or more datastores or memories associated with the one or more processors, and thesoftware may be implemented as one or more computer programs.Collectively, this special-purpose hardware and software comprises thefunctionality described above.

In the preferred embodiment, the functionality provided herein isimplemented as an adjunct or extension to an existing cloud computedeployment management solution.

While the above describes a particular order of operations performed bycertain embodiments of the invention, it should be understood that suchorder is exemplary, as alternative embodiments may perform theoperations in a different order, combine certain operations, overlapcertain operations, or the like. References in the specification to agiven embodiment indicate that the embodiment described may include aparticular feature, structure, or characteristic, but every embodimentmay not necessarily include the particular feature, structure, orcharacteristic.

Finally, while given components of the system have been describedseparately, one of ordinary skill will appreciate that some of thefunctions may be combined or shared in given instructions, programsequences, code portions, and the like.

Having described our invention, what we now claim is as follows.

1. A method, in a data processing system, comprising a processor and amemory, the memory comprising instructions executed by the processor formerging incoming alerts for accessibility, the method comprising:receiving a first input alert and a second input alert intended forpresentation by a screen reader; determining, by the processor, whetherthe first input alert and the second input alert arrived with aspecified time interval, wherein the specified time interval isdetermined so that the first input alert and the second input alert donot collide and have the second input alert interrupt the first inputalert; determining that the first input alert and the second input alertarrived within the specified time interval, operative to determiningwhether the first input alert and the second input alert are similar;determining that the first input alert and the second input alert aresimilar, operative to combining the first input alert and the secondinput alert in a merge operation wherein the output alert is a mergedalert, wherein the merge operation merges two similar elements whichoccur in each of the first input alert and the second input alert into asingle element; and sending the output alert to the screen reader forpresentation.
 2. The method as recited in claim 1, further comprisingprogram code operative to comparing dependency trees of the first inputalert and the second input alert to determine the similarity of thefirst input alert and the second input alert.
 3. The method as recitedin claim 1, wherein an equivalency rule is used for determining whetherthe dependency trees are similar.
 4. The method as recited in claim 3,wherein the equivalency rule is that a root node of each tree is thesame or equivalent and a specified number or percentage of the sub-nodesare the same or equivalent.
 5. The method as recited in claim 1, furthercomprising: program code, operative to reset a timer after a secondinput alert is received to determine whether a third input alert arrivedwithin the specified time interval of the second input alert so that thesecond input alert and the third input alert will not “collide” and havethe third alert interrupt the second alert.
 6. The method as recited inclaim 1, wherein the specified time interval is a function of a timeinterval required to present a first input alert.
 7. The method asrecited in claim 2, wherein the merge operation does not mergedissimilar elements which respectively occur in one of the first inputalert and the second input alert, wherein each of the dissimilarelements occur in the merged alert.
 8. The method as recited in claim 2,wherein a similarity rule is used to determine whether correspondingelements from respective input alerts are similar.
 9. The method asrecited in claim 4, further comprising a listener module whichintercepts alerts intended for the screen reader.